The development of complex military equipment traditionally has been based on a rigid, top-down approach, originating with a publication of a customer operational requirements document. The prime contractor decomposes the operational requirements document to allocate requirements at the weapon system level, which in turn are further decomposed and allocated at the subsystem and component level. In terms of system software, this decomposition is reflected at the software component level with components of the software being sourced/developed along the same lines as the procurement of the subsystems and components of the equipment. This top-down, hierarchical approach ensures that customer requirements are reflected in lower-level requirements and become integral to the objective weapon system design.
U.S. Application No. 20020150866 titled “Integrated evaluation and simulation system for ground combat vehicles” details some of the problems encountered by adopting such a rigid top-down approach. One of the problems encountered is that this approach does little to optimally allocate limited resources across a military system design, and objective characteristics of an operational design often exceed program constraints. In addition to sub-optimized designs, the top-down approach often leads to misallocated development resources and development processes that are incapable of rapidly responding to inevitable changes in operational, fiscal, and technological considerations.
Customer recognition of the above-described scenarios, the realities of tight fiscal budgets, and changes in the geopolitical climate during the past decade have had a noticeable effect on how future military systems can be developed and procured. For example, the development of modem military combat systems is cost constrained which could affect a system's capabilities. In addition, most forces are no longer forward deployed, but instead are forward deployable which translates into a need for combat vehicles that are highly agile, versatile and reconfigurable to rapidly adapt to changing battlefield situations. Inevitably, the ground combat vehicle system software that interfaces to and controls the operation of the combat vehicle subsystems is increasingly complex providing expanded capabilities to control weapon aiming and firing, navigation, power management, fault management, and communication. Additionally, once a component or system is deployed on the battlefield it is required to perform reliably and effectively which in turn significantly increases the performance expectations of the combat vehicle software system. This translates into little or no margin for software defects in combat vehicle software systems that are field deployed. Yet, the potential for defective software has increased with the increase in the complexity of the software.
The software development process continues to be dominated by a model in which components of the software are developed by different entities. Although software development processes are improving defect prevention and defect detection, measurement of software quality remains heavily dependent on expensive manual and repetitive software testing. In view of the above discussion, one of skill in the art will appreciate that updates to the combat vehicle software are inevitable, and each update increases the likelihood of software defects.
Prior art software development efforts for military systems have focused on the design aspect of software development. For example, U.S. Patent Application No. 20020150866 referenced above, addresses an evaluation and simulation system that functions integrally and interactively with the conceptualization, design, and development of ground combat vehicles, under conditions whereby design concepts can be analyzed, constrained resources can be allocated across a weapon system architecture in a manner that optimizes the weapon system's combat effectiveness, and a virtual representation of the weapon system can be tested under simulated combat conditions for combat effectiveness. Essentially, the design, analysis, and optimization necessitate the construction, coding and maintenance of complex algorithms to simulate the ground combat vehicles. However, the complexity of the algorithms makes it difficult to update the software that implements the algorithms without inadvertently introducing errors and makes it even more difficult to determine the effect of any introduced error.
Historically, software regression testing is conducted manually. When changes are made to the vehicle software, regression tests must be performed to ensure that the new changes do not produce unintended consequences elsewhere in the system. As vehicle software becomes more complex (in excess of 200,000 lines of code), software testing becomes more labor-intensive, which dramatically increases the cost for software testing activities. As a result, test activities are insufficiently funded which limits software test coverage.
The problem of software testing is exacerbated as development shifts from the waterfall model to the more robust iterative approach. With the waterfall model, regression testing is performed once, prior to the software release. With the iterative approach, regression testing is required after each software build. In complex systems, a software release can have multiple software builds that require multiple regression testing. This compounds the problem of expensive manual software testing. The prior art fails to address the problem of iterative application software testing in the context of complex military systems and subsystems where the yardstick is the reliable operation of the of military system controlled by the application software under battle field conditions.
Another example of software testing is described in U.S. App. No. 20040088602 which is directed to Automated recording and replaying of software regression tests. The application discloses a method and system of regression testing a computer software application in an execution environment wherein the software application interacts with data storages. In an exemplary embodiment, the software application is run a first time and interactions of the software application with the data storage are monitored. For example, the first output data written from the software application to the data storage are recorded, and input data received by the software application from the data storage are recorded. The software application is run a second time after the first time. While running the software application the second time, when the software application calls for data from the data storage, at least a portion of the recorded input data is provided to the software application, and, when the software application writes data to the data storage, second output data written by the software application are recorded. The second output data are compared with the first output data to evaluate whether the software application has passed the regression test. It will become readily apparent to one of ordinary skill in the art that the claimed invention considers output from a static data source instead of multiple, interconnected subsystems. In the latter case, the interrelationship between the responses, including the temporal relationship between the responses, takes on added significance. One of skill in the art would recognize that the regression analysis of the later case would necessarily have to take into account the interaction between the systems which is not necessarily captured by the methodology employed in the aforementioned invention.
Clearly, there is a need for cost effective but reliable software test and validation in a software environment that is becoming increasingly complex and where there is a fundamental shift toward iterative software development processes.